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FOREWORD 


The  theory  of  Markov  chains  is  steadily  gaining 
5.n  importance s and  numerous  new  applications  are 
being  made  of  this  theory * It  is  hoped  that  the 
results  here  presented  id.ll  be  of  interest  to  workers 
in  th©  field®  The  present  report  introduces  a n©Tf 
problem  in  the  theory  of  Markov  chains  by  considering 
three  states;  the  initial  state,  the  final  state,  and 
a taboo  state  which  must  be  avoided  in  the  transition® 
This  work  was  done  by  Professor  Kc  L®  Chung  while 
he  was  a guest  t^orker  at  the  National  Bureau  of 
Standards  during  the  summer  of  X9£lo  It  was  performed 
under  Contract  GST -52$  between  the  National  Bureau  of 
Standards  and  the  University  of  North  Carolina® 

J*  H*  Curtiss 

Chief,  National  Applied 

Mathematics  Laboratories 


iU  Ve  Astin 

Acting  Director 
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CONTRIBUTION  TO  THE  THEORY 


OF  MARKOV  CHAINS 


By 

Kai  Lai  Chung 


abstract 



sar-15,5  ??£ 

ST  n3W  ^3ults  obtained  LT  E°' 

Jiarris  turn  out  to  tie  -m  Wrv  „<•»/  J\..* 

amplifications  of  Dobli-'s  1 £ly  ?ith  80K® 

consider  the  probabilities  of  hin.?^  3 was  led  to 

before  another  J 1 htt,tinS  on®  state 

Idea  of  considehL  pi  r^a  thi-  **».  This 
•! . , « — o.ng  rnpoe  states  r>nft  in-?f  ?ei 

'taboo, n and  one  final  j,  »,!'  Ni?!  *ni\iel>  one 

the  present  work.  The  notion developed  in 

to  the  "union"  or  "interne -tier"  U pafsag®  ^me 

also  introduced  ker.h!  f two  state£  is 

notions  is  illustrated.  interPlay  between  these 


The  fundamentals  of  the  theory  of  denumerable  Markov 

ehaln“  Wlth  3tafcl°n£-ry  transltion  probabilities  (D.M.C.S.) 

-re  laid  down  by  Kolmogorov  [1]  and  further  work  was  done  by 

Doblin  [2],  The  theory  of  recurrent  events  of  Feller  [jj  is 

Cl°Sely  relatedj  if  not  ®o-extenaive„  Recently  some  interest- 

ir.g  new  results  were  obtained  bv  T p . » . _ 

v “c  ® Harris  [If.]  and  comrauni* 

c t o ci  go  the  author  rpv,..^  j. 

§ 

' :'jiphifi cations  of  Doolie j<>  umi,  <-> 

01121  - w«**  the  author  was  engaged  in 


1 n 


Cenurnersible !1  nean^i  , 

cu'  .n"  refers  to  a process  parameter 
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While  Harris*  main  purpose  lies  elsewhere,  he  was  led  to  con- 
sider the  probabilities  of  hitting  one  state  before  another, 
starting  from  a third  on©0  This  idea  of  considering  three 
(instead  of  the  customary  two)  states,  one  initial,  one  "taboo, 
and  one  final,  will  be  more  fully  developed  in  the  present  worko 
The  notion  of  first  passage  time  to  the  "union”  or  "intersection 
of  two  states  will  also  be  introduced  here®  The  interplay 
between  these  notions  will  be  illustrated® 

Recorded  results  in  this  paper  will  be  labeled  as  formulas 
and  theorems,  respectively e Relevant  remarks  as  to  their  origin 
or  significance  will  be  found  in  the  body  of  the  paper e The 
author  is  indebted  to  Dr®  Harris  for  communicating  some  of  his 
results  before  publication 

10  The'  sequence  of  random  variables  {X^}  n = 0,  1,  Z9 
forms  a D0M*C„S0  The  states  will  be  denoted  simply  by  th© 
positive  integers®  Tho  (one»step)  transition  probability  from 
the  state  i to  the  state  j will  be  denoted  by  pW  0 Because 
of  stationarity  we  have 

= J 

for  all  integers  m ? 0 for  which  the  conditional  probability  is 
defined o VJith  this  understanding  we  shall  permit  ourse^.ves  to 
write  xn  = 0 in  the  definitions  to  follow,  as  if  the  conditional 
probabilities  were  always  defined 0 
Notations o 

n,  H,  v,  r,  s,  denote  positive  integers  and  will  be  used 
as  time  parameters  or  general  numerals; 


' 


■ 


/ 


■ 


p 


it  j,  k„ & 


h„  denote  positive  integers  and  will  be  used 


as  state  labels: 

p}"i  = P(Xn  = j | X0  * i)  ; p(0)  = 

^■(n)  = P(xn  = j,  Xv  4=  k,  1 < v < n | Xq  * 

- P(Xn  a3,  XvH,  1 1 v < n | Xo  * 

£[f  - P(Xn  = J,  Xv  * J,  * k,  1 < v < n | 

00 

Q,*  = z oSn) 

n*l 


i + j 

i = j 


1) 

i) 


x0  = 1) 


where  Q may  stand  for  any  of  the  symbols  P 

k ij 


or 


P 

k ij 


We  offer  the  following  clue  to  the  above  notations 0 The 
letter  P designates  "passage;"  the  letter  F*  "first  passage;" 
the  f ir^st  right-hand  subscript  designates  the  initial  state; 
the  second,  the  final  state;  the  left-hand  subscript  designates 
the  n taboo  state,"  namely  one  to  b©  eschewed  during  the  passage 
(exclusive  of  both  terminals);  the  star  on  a letter  with  sub- 
scripts designates  the  sum  of  the  corresponding  infinite  series 
(finite  or  *oo)  summed  from  n “ 1 adc  inf a We  admit  that  this 
is  not  the  most  logical  system  of  notations  we  could  have  in- 
vent ado  For  instance,  we  have  the  superfluity  F(n)  =s  p(n) 

ij  j ij  * 

and  if  we  had  allowed  more  than  on©  left-hand  subscript^  , we 

could  have  used  only  one  letter  P and  written  F(n)  ® F(n) 

k ij  j,k  ij 


2 


This  naturally  suggests  the  consideration  of  more  than  one 
taboo  state 0 , < 


t 1 


' 


g 


CEJ 


However,  we  consider  our  notations  to  be  preferred  to  the 
arbitrary  use  of  all  sorts  of  letters  from  the  Latin  and  Greek 
alphabets®  Also,  after  painful  deliberations  we  decided  not 
to  define  P(^)  , Flo)  or  F(0)  , while  reserving  the  right  to 
do  so  later  in  some  cases® 

Formula  I®  If  i £ j*  then 

Ftr  iFij 11  * /IV  111 

where  on  the  right  side  0 9 00  is  to  b©  taken  as  0o3 
Proof®  Wo  start  from  the  formula 


/ x H-l 

p(n)  - 2 p(v)  F ( nc*v ) 

ij  v=0  3 ii  1 13 


(1) 


where  vre  agree  that  _,p(0)  = 1„  (1)  i3  proved  as  follows?. 

Either  the  state  i is  not  entered  at  all  during  the  passage  from 
i to  j9  which  contingency  contributes  the  term  corresponding  to 
v “ 0 on  the  right  side  of  (1);  or  there  is  a last  entry  of  i, 
occurring  at  the  step®  1 i v i n • 1,  which  contingency 
contributes  the  general  term® 

Summing  (1)  over  n,  we  obtain 

cd  n«l 

Z Z p(v)  p(n-v) 
n=l  v=0  j 11  i ij 

■ 13  i ijU  y ii ' » 

Since  the  terms  of  the  double  series  are  non®n©gative , the  in® 
version  is  justified  and  (I)  is  proved©  Moreover,  this  proves 

It  follows  from  (I)  that 

This  convention  will  b©  understood  in  similar  circumstances® 


1 t Ff, 

CD 

**  Z 

F(n) 

ij 

n~l 

ij 

CD 

QO 

e S 

p(^) 

2 

V^O 

j ii 

n«v*»-l 

that  if  ,F‘*^  > 0 , then  P*^  « qo 


■ 


. 


' 


* 


* 


' " j a ^ ^ an^  cn^r  = namely  p(n)  = 0 for  all  nQ 

Formula  II 0 if  j £ k 9 then 

■A,  - /r4  (i  * „ij’  • <”> 

Proof,,  We  start  from  the  formula 

P(n)  « 2 P(v)  P(n»v)  (2) 

k Ij  V=1  k ij  k jj 

where  as  before  p(0)  = 1.  If  v;e  ignore  the  left-hand  sub- 

k J j 

scripts  * (2)  reduces  to  a familiar  formula*  The  proof  of  the 
latter  extends  immediately  to  (2) « 

Summing  (2)  over  n we  obtain 


GO 


oo  n 


Pf  = 2 . P(")  =22  ,?(»)  p(n-v) 

* ij  n=l  k -*-j  n=l  v=l  K *j  k jj 

CD  00 

- Z f(v).  2T  p ( n-v ) , p*'n  + p*  ) 

v=l  ^ ^ J n=^v  k j j k i j ‘ k j j 

VI©  note  the  following  corollaries  to  (I)  and  (II).  to  bo 

used  later  0 

Formula  (Ila)*  If  i £ j then 


p‘iir  = F*'  r\  j.  p*f  \ 

i ij  i ij(1  iAjj' 

Formula  (lib).  If  j ^ k then 
P*  p*  m p*  p* 

j jk  krij  jk' k I j ° 

Formula  (lie)*  If  i r J then 

F*  (1  + P^  ) as  p*  (X  + P*  ) 
ir  i jy  i iju  j it ' 

Pornula  (III).  If  i f j then 

N 

2 p(n) 

lira  = P*  < 

■ l , . i ij  00  ° 

2 p(n) 

n=0  il 


(Ila) 


(lib) 


(lie) 


(III) 
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t i 


■■ 


Proof o W©  start  from  the  formula 


p(n)  ^ x p('?)  j>(n=v)  (3) 

U v*0  " 11  i iJ 

where  w®  agree  that  p(0)  a o0  Th®  proof  of  (3)  is  entirely 

1 i J 

similar  bo  that  of  (1)Q 

Summing  (3)  from  n®0ton~H9we  obtain 


N , , N n , x , . N , v N«v 

£ P £ 2 ?{<VMTV)m  2 p !)  2 O 

n-0  n^Ov^O  ii  i lj  v=0  -*-1  n*0 

W©  need  an  elementary  lemma  which  is  frequently  used  in 


(k) 


such  connections 


CO 


Lemma , Let  0 < av  < 19  by  > 0 ; 2 av  » 0 

v«0 

lira  bv  s b i 4oo  c Then 

N 

^ °N«v 

lim  *SQ  « B 

N-^oo  S 

^ Sr 

Applying  the  lemma  to  (1|.)  w©  obtain  (III)0  That 

clear  from  (II&),  and  th©  remarks  at  th®  end  of  the  proof  of  (X}0 
Theorem  1®  The  limit 

N 


. P ; , < 00  is 
i xj 


p(n.) 


lim 


ii 


H-^oo  H , , 

2 p(n) 

nio  JJ 


(5) 


exists  £.nd  is  equal  to  any  of  the  following  three  expressions: 


1 * J? 


r ii 


1 + p* 

i n 


F* 

. — u 

p* 

i ij 


p#  p# 

- LL 1 J-i 

p&  p* 

ji  i ij 


(IVa,b8o) 


( 


' 


. I ’ 


' 


the  first  always,  tho  second  if  1 - j.  the  third  If  F*  F**  > 0 

ij  ji 

Proof o Doblin  [2]  has  shown,  trivially,  that 


lim 
N co 


N 

% P(n) 
n-0  :i  2 
N “ " 

% p(^) 

n*=0  2 2 


o 


(6) 


Comparing  (III)  and  (6)  w©  obtain  (IVb)  if  i £ j0  (IVa)  now 

follows  from  (He)  and  obviously  holds  for  i = j*  If 

Ff  F*  > 0-  then  the  denominator  of  (IVc)  is  not  zero,  and  this 

ij  Ji 

is  then  equal  to  (IVb)  by  (lib)  with  k 32  i« 

That  the  limit  (5)  exists  and  is  finite  and  not  zero  was 
proved  by  Doblin  [2);  that  it  is  equal  tc  (IVa)  was  previously 
proved  by  the  author  [53©  The  present  approach  seems  to  be  the 
simplest o 

2S£2HS£2.°  If  I?,  J,  arc  distinct  states  of  one  class s4 


thon 


N f 

S pfn) 

lim  BSLdL 

M-»ao  ® 

X P,'n/ 
n-1  k£ 


a 


Naturally  there  are  other  expressions  for  it  and  we  omit 
tho  tedious  considerations  when  some  of  the  states  are  identical* 
2*  We  now  consider  two  states  i and  j belonging  to  tho  same 
r o c ur rent  c 1 a s s , name ly : 


7?# 

" ii 


ss 


* 1 


© 


r; 


f Slightly  generalizing  Kolmogorov,  we  define  a class  to  be  a 
sot  of  state 0 such  that  for  any  two  states  i and  j belonging 

to  it,  wo  ha vo  Pf  p*  > 0o  See  [6]0 

ij  jl 


c: 1 
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J ' 
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■ 
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A fundamental  idea  in  the  theory  of  D#H0C0S6j  already  found 
in  JLoImogorov!s  work*  is  to  notice  that  whatever  transpires  between 
successive  entries  ab  a recurrent  state  forms  a sequence  of  in- 
impendent  events©  F.irsuing  this  idea,  Harris  [43  discovered 
the  following  very  elegant  theorem©  Our  proof  is  somewhat 
different  from  his© 

Let  i i j and  define 

Yn  = uhe.  number  of  v,  1 < v in,  such  that  X7  «*  i; 

Zn  th)  number  of  vp  1 < v i n,  such  thit  X7  * j0 

In  words , Y (or  t ) is  the  number  of  entries  at  the  state  i (or  j) 
n a 

in  the  first  u at3ps0  Using  the  average  ergo&ic  theorem  (see  (11) 

(• 

below)  it  ie  jaay  to  show  that^J  la  a positive  state?  and  P(XQ®CJ  * 1 
where  C Is  Ae  class  containing  then  we  have 

Z. 


P(  lim 

n-€>oo  & p iv 


1)  •»  1 


2l 

The  folic j is  theorem  covers  both  positive  am  null  classes 0 

?h>  rtm  2 (Harris)©  If  i and  j are  two  ;<;;at©a  in  a recurrent 
class  C i'd  ?(Xq6G)  .«  1,  then 


n , X 

3 p(v) 

pf  s lim  v—o  \ & 

‘n*oo  Yn  n»oo  n ~~  ' 

£ p(v) 
v*0  ii 


(7) 


P.oof  © Since  i is  recurrent-  we  have  P(  y«  *•  oo)  = 1 

n«*co  n 

Let  v j *•  Vo  <**©  be  the  successive  indices  v,  such  that  *v  * * 
Let  fl*  » the  number  of  v0  < v < suet  that  .lr . « 

Tr.cn.,  r remarked  above,  the  Wg9fi  are  independent,  identically 
di:-f  1 hated  random  variables©  Evidently  we  have 


■ 

1*1  \ : V 


>.\  i 


• ■ : 

oc-  b s - wc-i  a o*  (.8; 


t - * ;v  r 3 x‘ ; J ilriB^xioc  eaalo 


a q rtf  ©si  el©* 


’ * 

y ■-  an  i w t.  *3  -’  * -:  - • 


c>7.  eec  ;>Oi/o  dcV  *>d 

v > av  to  aetfmwr.  i 

.7  •;  ' ■'  ■ ' £> 


TO  ^ = 


E(Vto ) 


CD 

2.  4^ 


- P#  < OD 


n»l  i 13  " i U 


?!: *.  tj  definition  we  have 


vYr  - 


“ * VYn  * 1 


and 


*n~i 

Z Wo 
S'l 


•a 


i«si 


V1  ^ ^ ws 
a=l 


Consequently 


Yirl 
£ W; 
S K 1 

"TT 


Y, 


Zn  Tl 

^ YTiE^ 


n 


Ti 


■n 
2 ws 
8-1 


-n 


(8) 


Applying  Khintahine-Kolmogorov3 a strong  law  of  large  numbers 
so©  @eg0  [9]  pog08)  to  the  sequence  {i K we  obtain? 


•n 


p f 11m  S=1 


<£ 


i «s 


♦oo 


pw  ) = 1 


Yr  1 ij 

Moreover*  P(vi  < 4-oc)  = 1 „ It  follows  from  (8)  and  (9)  that 


(9) 


PClim  * p*  ) - 1 

n-soc  Yn  1 ij 


(10) 


Now  Ff  =s  X0  Hence  Theorem  2 follows  from  (10)  and  Theorem  1* 

1 j 

using  ( IYb ) there© 

This  theorem  Includes  as  special  case  a previous  result  by 
Erdos  and  the  author  17]*  Consider  independent*  identically 

distributed  random  variables  {Un}  which  assume  only  integer 
values  x^rlth  mean  zero©  They  form  a DoHo0oS©  with  all  integers 

as  the  states  © Since  the  mean  is  zerq,  all  possible  states  are 
recurrent  by  a theorem  of  Fuchs  and  the  author  [8]©^ 


This  important  step  cannot  b©  circumvented  by  the  present 
more  general*  approach*, 


c p » 

:c-  ...  ' r 

, > - 

v'< 


* 


:;1  1o  wal  susa-s^a  a * :#oia%oiaX«au*oJrio*«*Jtt 

; ■ ' ’ M)  eoi'.f?  up " 


■ 


. ■ 

' 


10 


Without  loss  of  generality  w©  may  suppose  that  every  integer 
is  a possible,  therefore  recurrent,  state 0 Writing 


m 


Sn  - U Uy  we  see  that 
v»l 

P(n)  * p(sn  = j-i) 

Hanca,  F;*1)  * = P „ and  (7)  becomes 


H 


00 


7 

which  is  Theorem  8 in  JiS]o  Needless  to  say,  as  far  as  this 
statement  is  concerned,  Harris0  approach  is  incomparably  better « 
However,  we  note  that  there  we  actually  proved  a sharper  result, 
i o e o 


l*e 


P Mj}  T loOe) 

xn 


0 


n 


for  every  e > 0,  where  Mn  * $ ?(SV  « i)  0 see  also  Theorem  ? 

ytsl 

1 


in  Ip]  o It  would  be  of  interest  to  investigate  eorrespond5.ng 
strong  relations  for  the  general  Markov-chain  case,  using  perhaps 
& more  precise  form  of  the  strong  law  of  large  numbers  G 

3o  W©  now  consider  a positive  recurrent  class  Ce  Accord- 
ing to  Kolmogorov,  in  C all  mean  recurrence  and  first  passage 
times  are  finite,  namely  for  all  i,  jeC  we  have 
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V®  introduce  the  notions  of  first  passage  to  j(Jk  and  to  as 

follows©  Let  j # k0 

T(i  ,jOk)  * the  smallest  integer  n > 1 such  that  Xn  = j or 

52  kp  whichever  happens  first*  given  that  Xq  * i; 
T{i?j$k)  = the  smallest  integer  n > 1 such  that  there  exist 

. two  integers  nq  and  ng  such  that  n^S  g 1 1 i n* 


Thus  ©•■  a the  event  that  Xq  **  i and  the  state  j is  reached  before 
the  stat'.i  k©  Sine©  I*  j and  k belong  to  one  recurrent  ©lass 

we  have 

■'}  ©k  « {ws  Xq(w)  « 1} 

Let  P(Xq  38  i)  ~ c > 0 © W©  have  th®  following  relations*  immediate 
consequences  of  the  definitions© 


»(i,J«k)  = E{T(i,j©k)}  ; 

m(ijf)k)  - E { T(i,  J©k)  } 


Let  w cle  iote  the  sample  point  © Put 


©ji  :s  { w i. Xq(w)  * i,  $ j*  f k for  1 < v < n;  X^v)  =>  j } ? 


* Jti{  I j nP(dw)  *Jk  <n+BSkj)  M(u)  ) 


* C3n(i, jfik)  + p(e*£)  mk< 
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cm>:  , j$k)  ~ ^ lj  ^n*mjk)  P(dw)  + f (n+n^j)  ?(dw)  } 

n~x  < * ok 

n n 

- cm(i9jOk)  + P(©3)  raj^  + P(e^)  o 

Now  by  definition  we  have 
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k ij 


z<j£l 
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pfc 

j Ik 


Hence  v®  obtain  from  the  above; 

Fc.?t  ala  VI  o If  J ^ k then 

m(1,  Jtnc)  * m13  - jPik  irijjj  * mlk  • i^ij  Mjk 
Fcrr  ula  VII®  If  j ^ k then 

ra(i,J$k)  * Hij  + kpij  «jk  “ ®ik  + jpik  Mfcj 

Sine  e 


(VI) 


(VII) 


p#  4.  F*  _ 1 

k ij  j ik  1 
we  deduce  from  (VII): 

Fcrrnla  VIII.  If  j ^ k then 

raik  + mkj  - mij  * k^ij^jk  + »kj>  • <vm> 

i of  (VIII): 

mkk  * kFkjf-mjk  + mkj)  « (Villa) 

This  las  formula  is  due  to  Harris  [Ij.],  who  also  derived  from 
It  the  following  cute  relation: 


We  oote  the  following  special  case  (i  = 


k^kj 


. (VUIb) 

% 

Now  in  a positive  class  the  ergodic  theorem  of  Kolmogorov  holds 


& This  theorem  actually  establishes  the  limit  of  p(n)  as  n^ooc 

The  average  form  (11)  is  an  easy  consequence  of  A^Hardy® 
Littlewood  Tauberian  theorem! 


VO 
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1 $ p(v) 
oo  n * A m m. 


(11) 


v=l  4,1  jj 

Thus  (VUIb)  turns  out  to  be  a special  case  of  Theorem  1,  using 
(IVc)  and  noting  that  = 1 G 

Dividing  (VIII)  by  the  product  of  (Villa)  and  (VUIb) 
wo  obtain 


mlk  » mkj  . n13  , 

■ — wnm  ■ nmwiurtli 

m a 


,IBr"  'L1"' 

y 1* 


(12) 


By  Formula  (lib),  the  left  side  is  , since  = 1 in  the 

present  case®  Thus  we  obtain 

Formula  (IX)*  If  J ^ k then 


p*  - mlk  + mkj  ~ mij 
k iJ  * rajj 


(IX) 


As  an  application  consider,  as  in  the  Central  Limit  Theorem 
for  Harkov  chains,  random  variables  {Yn}  attached  to  the  Markov 
chain  (xn  \ in  the  following  wayr 

Y ■ x,  if  X * i where  the  x, ®s  are  arbitrary  real  numbers 
n i n 1 * 

Theorem  3.  Let  1 be  a positive  state.  Given  XQ  « i,  let 

v^  denote  the  smallest  n > 1 such  that  X * i_  Then  if  the  series 
u n 

on  the  right-hand  sides  converge,  we  have 

® r:  i n 

E {Y^  ❖ ooo  + Yv  | Xq  ~ i } 55 

E {(YX  ♦ 
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— - ■*•  2:aii 

y .1 

df 

n3S 

j=*i  njj 

k-1 

(Xa) 

(Xb) 


raji  + mijf  • mjk 

mkk  * 


j>oob  pointed  out  to  me  that  this  is  a case  of  ftol&8s  equation 
for  Markov  chains 0 


Jt ; ! t f : 

" 1 


. 


-©fosfra  ;vJU  Lsoq  a od  1 NHL 


' 

. 


’■ 

■ 


t 


114-  - 


Proof o It  Is  more  convenient  to  consider  new  variables 
Zn  defined  as  follows: 

( 0 if  Xy  s i for  some  ve  1 * v < n | 

2 » <i 

£ x.  If  Xn  = 3 and  Xy  >j=  i,  1 i v < n ; 
inhere  j may  be  I in  th©  last»writtan  lin©0  Evidently  we  have 

then 
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bjj 

by  (IX)  with  i * k. 

Fur (her more  we  have 
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E {(  2 YxO^IXq  - 1}  - E {(  Z Zn)2jxo  = i } 
n«l  n~l 


oo 


* E { S ( Z~-  ■»  2 £ Zp  Z&) 
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As  before  we  obtain  readily 

co  , oo 

E { 2 z2  } . * !ii  *2 
n=l  mjJ  J 

Next  we  have 
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£ 2 £ £ 4p|?)  X,  1P(S“p)  X. 

r*l  s*r+l  3*1  k=l  1 1J  J 1 Jk  k 

j+i 

CO 


By  (IX)  this  reduces  to  (Xb)0 
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The  two  expressions  on  the  left  sides  of  (Xa)  and  (Xb) 
play  an  Important  role  in  Dobiin ‘ a Central  Limit  Theorem  for 
Do^^oSo  Vo  refer  the  reader  to  [2]  for  details.  They  are  here 
evaluated  in  what  seems  to  us  more  tangible  terms, 

I4.0  From  Formulas  (VI)  and  (VII)  it  follows  that 

v 

€1 

This  relation  ia  in  striking  resemblance  to  a familiar  formula 
in  the  elementary  calculus  of  probabilities,  according  to  which 
If  A and  B are  any  two  events  then 

P(a0B)  + P(A$B)  « P (A ) + P(B)  o 


The  generalizations  of  the  last  relation  to  any  finite 
number  of  events  i3  known  as  Poincare 8 s formula  (see  8sg0  { 9 j 9 
p0  61);  and  we  immediately  suspect  that  the  same  may  be  true 
for  the  msan  first  passage  times 0 This  is  indeed  so,  W©  define 
o»oUh)  and  0../lj0)  as  the  obvious  extensions 

from  tha  case  3*2.  Wo  shall  also  rcrite  t#0fl  38t  J to  de- 
note ij-n  0#or*  jr-ln  jr4>1n  if  j • jr  u<r<s)  and 

4,n  ,,,nj  is  not  one  of  the  j 3s. 
is  r 

Formula  XI.  If  »o«#  Jt  are  distinct  states  In  a positive 
class  to  which  i also  belongs,  then 

»(*,&».  = Sm(i,jr)  » 2 m(i, jr,Ojro) 

r~l  lii‘-,<r2<8  1 


II 
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+ (-1)8"1  ...  $ja)  . 


(XI) 
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Proof  e Put 

'n  “ (vt  *<}(«)  35  i»Xv(w)  4=  Jip  ..o8  j8  for*l<  v < n,  xn(w)  * jp  } 
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#a  haV0'  as  at  tte  *S Inning  of  § 3, 
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1 3 r-1  _ ^ 

Substitute  (13)  int0  the  -i«ht  b1(1a  A , ? 

t»l«a  c-lpur,.,;.  . , 8 S1“  * •»*  •«■««  the 

A f ] 0 0 0 


? ?1 


,}  wh3r®  r * « diet  inct  from 


* *,  T.  . r diet, 

U Sppears  311  right  side  of  Ul)  Onoe 

“<iBV  " V and  once  In  ®(ieJ  ^ « n 

Wito  eigne;  and  dees  not  appear  in  **  " * ’ 

D8t  oa  the  right  aide  of  (i^V^  ***""*  ^ ** 

to  consider  the  term  «(*,<„_  _ S"r°0  " r<SmalllS 

term  and  hence  it.  net  coefficient  lY  “ * ‘WS  *“*  * eVdry 

'1^  " !J)  + (|)  + + (=i)S~l  a 1 

Therefore  (XI)  i.  establish. 
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- ~ w vJZZZZ  7"’  '~U  “ » 

«ny  closer  relation  bafcw&*n  *u 

P^ent  twins.  We  aiso  We  , * the  two  ap- 

Jcnown  extensions  of  Polo  '•  ’ " °*  ®nsiona  »ugge8ted  by  the 

Poincare  formula  to  the  interested  reader 

•'*'  We  now  giV0  another  ethod  nf 

-et.nod  of  computing  p*  ~ , 

met  nod  requires  the  ergodie  - « ' ' 1Ms 

8 dlc  J;eorenl  (ID.  An  interesting  h 
proauct.  is  the  following-  * testing  by- 
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Theorem  5>0  If  i9  j and  k (not  necessarily  distinct) 
belong  to  a positive  class s then 


ao 


2 { FI")  - pte}  } * o 

n=l  1K  iA  ^ick 

Pro<fe  Using  the  familiar  formula  (cfe  the  remark  after  (2)) 

p(n)  * 5 p(v)  p(n-v) 


ik 


v=a 


Ik  kk 


w©  h&vt 


- ?ss!  > ■ X{T&'  - pS’ 1 5 p^’  • aw 

Substitut  ing  from  (XI)*  we  see  that  the  right  side  of  (lij.)  Is* 
ae  N-hjo,  asymptotically  equivalent  to 
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v— 1.  mkk 


(15) 
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Using  this  in  (15) » we  see  that  its  limit  as  H ^ oo  is 
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We  rote  that  Theorem  5 gives  a convenient  determination  of 
the  mean  first  passage  times  in  terms  of  th©  transition  proba« 
bilitiesi  in  particular 
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nuk  * 1 + 2 { p£^  ° P(-?)  } „lim  i.  2 P,(”) 

We  do  not  know  what  the  situation  is  in  a null  class 0 All  we 
can  info?  from  Theorem  1,  (IVc),  las  If  i and  j belong  to  on© 
recurrent  class  and  4s  ^hen 

oo 

S (»!})  - p$3>  J 


— CO 


To  ?©turn  to  . P 


ft 


k ij 


If  J 4s  k we  have  evidently 


1 , p(p)  p(D  « p(n+l) 

h*k  k ih  '*$  k ij 


(16) 


where 9 as  later,  an  unspecified  summation  runs  from  1 to  oo  0 
Summing  (16)  over  n,  we  obtain 

% p#  1)  3 p#  « p(l) 

h|k  k ih  hj  k i j k i j 
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since  , » 1 and  . pj^)  « pj^) 

k ik  k ij  ij 

We  is sent  that  In  general 
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1 ,F*  Pi”>  - .P*  + 2 {P.£)'  - p|v)>  . 

h k ih  hj  k ij  V=1  kj  ij 

This  is  readily  shown  by  induction  on  n9  starting  with  (17) o 
Now  sum  ?rom  n~l  to  n=N,  divide  by  N9  and  let  N’=^co  0 By  (11) 
and  Theorem  5 we  obtain 
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(18)  ani  (19)  give  { X) « 
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